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Abstract 

The paper presents a practical approach to measuring liver motion, both respiratory and laparoscopic, with a tool 
guided in the operating room. The presented method is based on standard operating room equipment, i.e. rigid laparo- 
scopic cameras and a single incision laparoscopic surgery trocar. The triangulation algorithm is used and stereo cor- 
respondence points are marked manually by two independent experts. To calibrate the cameras two perpendicular 
chessboards, a pinhole camera model and a Tsai algorithm are used. The data set consists of twelve real liver surgery 
video sequences: ten open surgery and two laparoscopic, gathered from different patients. The setup equipment and 
methodology are presented. The proposed evaluation method based on both calibration points of the chessboard 
reconstruction and measurements made by the Polaris Vicra tracking system are used as a reference system. In the 
analysis stage we focused on two specific goals, measuring respiration and laparoscopic tool guided liver motions. We 
have presented separate examples for left and right liver lobes. It is possible to reconstruct liver motion using the SILS 
trocar. Our approach was made without additional position or movement sensors. Diffusion of cameras and laser for 
distance measurement seems to be less practical for in vivo laparoscopic data, but we do not exclude exploring such 
sensors in further research. 
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Introduction 

Nowadays one of the most important directions 
in radiology and surgery is computer-aided diagnosis 
and therapy [1]. As regards computer-aided surgery, 
the research is focused on image-guided procedures. 
In order to construct the navigation system, informa- 
tion on movability is required, especially as regards 
the soft tissue in the abdomen. 

Our objective was to develop a non-invasive 
methodology of liver surface points detection and 
tracking. There are many approaches relating to com- 



puter-assisted surgery, e.g. laparoscopic teleoperat- 
ed robot-assisted surgery, da Vinci, image-guided 
laparoscopic surgery, and laparoscopic surgery com- 
puter simulators [2, 3]. Our approach was to use stan- 
dard equipment (laparoscopic cameras, typical surgi- 
cal tools), and not interfere with medical procedures. 
We also planned to use our approach for both laparo- 
scopic and single incision laparoscopic surgery [4, 5]. 

While developing the methodology we took into 
account existing approaches. Many works have been 
done in beating heart surgery. Bogatyrenko et at. pro- 
posed a stochastic physical model of the beating 
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heart surface [6]. Acquisition was done with a trinoc- 
ular camera system [7]. Cortadellas et ai proposed 
the concept and overview of an augmented reality 
system that assists laparoscopic surgical procedures 
[8]. Mountney et ai proposed stereo cameras, a com- 
plex anatomical structure movement based on a mod- 
ified Bouguet algorithm and used phantom tissue to 
model 3D geometry [9]. There also exists heart motion 
estimation and prediction using a Fourier series mod- 
el, a vector autoregressive model and the extended 
Kalman filter. Groeger et ai proposed a laparoscopic 
tool tracking colour base segmentation and solved the 
problem of deformation and illumination occlusion of 
objects [10]. Chen et ai presented a feasibility system 
for guiding the robotic removal of the prostate that will 
provide augmented reality surgical guidance [11]. 
A stereoscopic fibroscope imaging system for minimal- 
ly invasive surgery (MIS) is presented in [12]. The major 
goal was to design flexible instruments that can follow 
curved anatomical pathways by using the Simultane- 
ous Localisation and Mapping (SLAM) algorithm. 

An overview of the literature shows that the data 
about liver motion for a specific patient are not 
known so our future development work is regarded 
by us as a milestone. In the results of our work we 
want to use in real time a non-rigid registration algo- 
rithm of liver motion. This method focuses on simple 
algorithms which can be used in deterministic short 
time and are applicable to an online processing 
approach with a constant constrained frame rate in 
consultation with the surgeon. 



Aim 

The paper presents a practical approach to meas- 
urement of liver motion, both respiratory and laparo- 
scopic, with a tool guided in the operating room. 

Material and methods 

Our approach can be divided into four steps: 
equipment setup, camera calibration, selecting the 
3D reconstruction algorithm, selecting the triangula- 
tion points. 

Equipment setup 

The first and most important principle of our 
approach was to develop a non-invasive methodolo- 
gy (Figure 1). Laparoscopic cameras are nowadays 
standard equipment in the operating room, so we 
have focused on computer vision algorithms and we 
need at least two cameras to obtain a pair of stereo- 
scopic images. However, there are some monocular 
reconstruction algorithms but there are many limita- 
tions which are difficult to take into account during in 
vivo laparoscopic reconstruction, e.g. requirement of 
a preoperative 3D model and ambiguity of depth map 
reconstruction. In order to apply a computer vision 
paradigm we need to mount the cameras together 
rigidly, so we prepared a special connecting device as 
seen in Figure 2. To set the system up we use two 
complete laparoscopic vision systems: Olympus 
CV-180 video processor HDTV, CLV-180 Light Source 
and an "Endo-Eye" laparoscopic camera with a Matrox 




Figure L Setup diagram 
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Orion HD frame grabber. One important part of the 
setup is to synchronize the video stream. Generally 
trigger input is not included in the laparoscopic video 
acquisition system. This was the reason we needed 
to develop a software camera synchronization solu- 
tion. This is based on an application with two back- 
ground threads, and to balance them to record equal 
frames on both cameras. We use a global counter in 
the main thread to mark the frames. For laparoscop- 
ic surgery we use an Olympus LESS trocar. 

Camera calibration 

In our research we tested two calibration meth- 
ods, the Tsai algorithm [13] and the Bouguet [14] 
method, both with satisfactory results. The Bouguet 
algorithm is based on a simplified pinhole camera 
model and low computation requirements. The Tsai 
method is more complex and can be divided into two 
steps: calculating the pose (3D orientation, andx-axis 
andy-axis translation) in the first stage, and comput- 
ing the focal length, distortion coefficients and the 
z-axis translation in the second stage. We use the cal- 
ibration grid and OPENCV Library (Hough Transform) 
to get sub-pixel coordinates of corresponding points 
and calibrate the camera with the Roger Y. Tsai algo- 
rithm [15]. For calibration we used two perpendicular 
calibration grids (chessboards). We gained the best 
results when the number of calibration points was 
greater than 30. 

Selecting the 3D reconstruction algorithm 

There are many approaches to the 3D reconstruc- 
tion problem, but there are many limitations to its use 
in the laparoscopic reconstruction. Difficult cases with 
shape from shading (SfS) need multiple light sources 
[16, 17]. Because of liver surface reflection we use only 
one light source and the SfS is not practical for our 
approach. The texture of the liver surface is not distinct, 
so we also reject the shape from texture (SfT) [18] 
approach. After evaluation of different reconstruction 
approaches, we decided to use the classical triangula- 
tion technique. Despite the imperfections of the trian- 
gulation method, such as sensitivity to illumination and 
artefacts that cause pixel occlusions, we decided to 
adopt this method and prove its utility, and obtained 
high accuracy. Our approach is more challenging than 
one by attaching landmarks or colour markers to the 
liver surface. Triangulation x is based on finding the 
position of a point in 3D with known camera matrices 




Figure 2. Stereo camera setup 




Figure 3. The geometric interpretation of a trian- 
gulation of point X with known cameras C, C 

and projections of a point into both views (Figure 3). 
There are two types of estimation assessment: root- 
mean-square (RMS) residual error (distance of the mea- 
sured and the estimated value) and RMS estimation 
error (distance of the estimated and the true value) 
[19]. We focused on open surgery, but our approach 
was flexible and could be used in laparoscopic surgery. 
Two rigid laparoscopic cameras were used. 

Selecting the triangulation points 

As regards state of the art 3D reconstruction algo- 
rithms, the manual finding stereo correspondence 
seems not to be the perfect approach, but because of 
nonlinear shape changing in different segments of the 
liver, we decided to use it. Our main motivation was to 
verify the general knowledge of liver motion and collect 
specific knowledge to propose the proper shape de- 
formation constraints to deploy the monocular shape 
from a motion approach (SfM) to the liver surface (bas- 
ed on standard laparoscopic equipment) [20]. Our pre- 
liminary approach was to use well-known solutions, i.e. 
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Figure 4. Picture of the liver during surgery 

RANSAC and Lucas-Kanade [21]. We tested Image and 
Video Processing - MATLAB and we were not satisfied 
with the RANSAC results. Secondly we evaluated the 
Lucas Kanade method in OPENCV for the optical flow of 
the liver. The reasons for the failure of both methods 
were lack of corners and edges, and a modest gradient 
on the limited visible part of the surface. 

The fundamental problem of using the RANSAC 
algorithm and the other method (optical flow) for liv- 
er images is that the liver is not a rigid body. It is dif- 



ficult to have a basis for the results obtained auto- 
matically for three reasons: 

• ruffles on the surfaces (some points of the surfaces 
disappeared in the frame sequence), 

• specific lightning caused by abdominal characteris- 
tics and the construction of the laparoscopic camera 
(after the pilot study we used only one light source 
to minimize the reflection of light effect), 

• liver deformation (the distance between specific 
corresponding points in the frame sequence is not 
constant). 

Our data set consisted of 10 stereo video sequen- 
ces for open liver surgery procedures. We also record- 
ed 2 laparoscopic procedures using the SILS tech- 
nique. After obtaining the videos from the cameras 
we tried to select corresponding frames using the 
global counter (Figure 4). During the video acquisition 
we used two balancing threads to grab frames from 
two cameras and mark frames with the value of the 
global counter. The steps of preparing data for the tri- 
angulation algorithm were: 

• select the corresponding frames based on the glob- 
al counter values, 

• mark the characteristic points. 

In order to assess the intra-operator variability in 
points of interest (POI), the manual clicking of the POI 
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Figure 5. Zoomed pixel values - imshow and local extrema - red rectangles 
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positions on selected stereo images was repeated 
twice by two operators. We tracked from five to 
ten POIs on each frame which were localized on spe- 
cific liver segments. Basically we tried to use anatom- 
ical landmarks, but it was difficult, or even impossi- 
ble. Therefore we decided to choose local extremes 
(Figure 5). 

In practice, we used imtool and imshow standard 
MATLAB image analysis tools (Figure 6). We also tried 
to use colour base segmentation. The motivation was 
that the colour of standard laparoscopic tools (silver or 
black) is different from the liver surface colour (hue of 
brown). Using this approach it was possible to find the 
contours of the laparoscopic tool, but impossible to 
find specific points. Because of our principle of using 
specific marked points, we discarded this approach. 

The proposed evaluation method based on both 
calibration points of the chessboard reconstruction 
and measurements made by the Polaris Vicra track- 
ing system are used as a reference system. 

Results 

We have presented separate examples for left 
and right liver lobes (Tables l-ll). These results were 
obtained by performing the triangulation algorithm 
on manually found points in corresponding video 
frames. The first six left columns present the two dif- 
ferent position of the same POI in a world coordinate 
system (measured in mm) in video sequence, the next 
two RMS residual error of reconstructed positions, 
and the last three the displacement vector. We ob- 
tained on average 3-4 mm RMS residual error with 
manually selected corresponding POIs. Due to van- 
ishing pixels in the video sequence, RMS residual 
error of some POIs is up to 10 mm. 




Figure 6. Picture of the liver with the selected 
points of interest - imtool 



Discussion 

Generally the results confirm the expectations of 
the scope of liver movement [22, 23]. Respiratory 
motion is one rank lower than the possible motion 
guided by the laparoscopic tool. It is possible to 
reconstruct liver motion using the SILS trocar but it is 
more complicated to set up the equipment using two 
single view cameras than with open surgery. Using 
one stereo camera (e.g. da Vinci Surgical System) will 
be a great simplification as regards camera calibra- 
tion and mounting the camera in the SILS trocar, but 
this is not commonly accessible in operating rooms 
nowadays. 

Finding the correspondence points manually 
requires deep image analysis, which allows observa- 
tion of the wrinkles of the surface of the liver. Some 
points of the surface disappear in the frames se- 
quence. It creates a challenge to use monocular SfM 



Table I. Sample data for right lobe laparoscopic tool guided motion 



Initial position X{ 


Final position X" 


X{ RMS 


X" RMS 


Displacement AX, 


X' 


r 


T 


X" 


Y" 


Z" 


error 


error 


AX 


AY 


AZ 


-371.12 


134.23 


-292.83 


-362.02 


115.8 


-278.2 


4.33 


3.32 


9.1 


-18.43 


14.64 


-380.91 


137.67 


-300.91 


-371.59 


119.57 


-282.64 


4.33 


3.41 


9.32 


-18.09 


18.27 


-376.38 


139.8 


-289.03 


-370.87 


116.39 


-276.36 


3.79 


4.22 


5.51 


-23.41 


12.67 


-366.92 


129.6 


-287.85 


-366.0 


112.95 


-275.24 


4.08 


2.82 


0.92 


-16.65 


12.61 


-365.85 


128.02 


-288.77 


-369.23 


112.35 


-282.0 


4.12 


3.06 


-3.38 


-15.67 


6.77 


-360.16 


129.03 


-293.28 


-361.99 


112.16 


-275.39 


3.98 


3.09 


-1.83 


-16.88 


17.89 
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Table II. Sample data for right lobe respiratory motion 



Initial position X{ 


Final position X" 






Displacement z\X< 


A 


V 


7' 
z. 


Y" 

A 


V" 


7" 


error 


error 


A V 
\ A 




A7 


-379.29 


119.29 


-306.6 


-375.94 


121.42 


-305.09 


2.73 


2.17 


3.36 


2.13 


1.51 


-354.81 


124.02 


-274.31 


-350.31 


126.84 


-272.28 


3.96 


3.15 


4.5 


2.81 


2.03 


-472.88 


106.5 


-455.97 


-475.86 


105.01 


-456.23 


13.43 


13.75 


-2.99 


-1.49 


-0.27 


-490.03 


97.89 


-460.29 


-493.59 


96.24 


-460.85 


12.67 


12.78 


-3.56 


-1.65 


-0.56 


-490.25 


100.01 


-440.12 


-492.79 


98.54 


-440.94 


9.48 


9.37 


-2.54 


-1.47 


-0.82 


-450.27 


111.73 


-414.62 


-451.37 


111.06 


-415.04 


9.8 


9.71 


-1.11 


-0.67 


-0.42 


-450.19 


111.06 


-415.66 


-451.36 


110.35 


-416.09 


8.96 


8.87 


-1.16 


-0.71 


-0.43 


-445.63 


111.32 


-414.03 


-446.61 


110.8 


-414.27 


7.83 


7.84 


-0.97 


-0.52 


-0.24 



reconstruction even to find the correspondence 
points automatically. 

The presented pilot study triangulation approach 
to liver surface tracking with manually selected corre- 
spondence based on two standard monocular laparo- 
scopic cameras needs improved accuracy to allow 
medical application, e.g. tracking respiratory motion 
during radiotherapy. We are planning our future work 
in two directions. One is to find correspondence 
points automatically with sub-pixel accuracy. There- 
fore we will try to use a specific Olympus camera nar- 
row band imaging which is the camera specific mode 
to enhance the vessels to aid finding the POIs. The 
second is to try to use only one camera for monocu- 
lar surface reconstruction. 

Our pilot study is a preliminary stage of develop- 
ment of a navigation system for treatment of unre- 
sectable liver tumour. The navigation system allows 
the surgeon to track the current position of surgical 
tools as regards the positions of anatomical struc- 
tures during intervention; therefore tracking move- 
ment of anatomical structures is required. There are 
a few methodologies to find a solution. Bogatyrenko 
etal. [6] proposed a stochastic physical model for the 
similar problem of beating heart surface tracking. 
Mountney et al. developed stereo cameras, a com- 
plex anatomical structure movement based on a mo- 
dified Bouguet algorithm and used phantom tissue 
to model 3D geometry [9]. Groeger et al. used a lap- 
aroscopic tool tracking colour base segmentation and 
solved the problem of deformation and illumination 



occlusion of objects [10]. All of them need special 
equipment to obtain positive results: stereo camera 
[9], colour laparoscopic tool [10] or complex mathe- 
matical models [6], that need to be adjusted to a spe- 
cific patient. We have proposed a methodology based 
on standard equipment and does not require com- 
plex mathematical calculations. A disadvantage of 
our method is extracting correspondence POIs manu- 
ally, but it could be overcome by using specific POI 
image descriptors or applying the monocular shape 
from motion algorithms. 
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